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Abstract 

Combinatorial S n -analogues of Schubert polynomials and corresponding symmet- 
ric functions are constructed from an exponential solution of the B n - Yang-Baxter 
equation that involves the nilCoxeter algebra of the hyperoctahedral group. 



1. Introduction 

Schubert polynomials of Lascoux and Schiitzenberger (see [L, M] and the literature 
therein) can be introduced and studied from different points of view. In this paper, 
we adapt to the B n case the approach to the theory of Schubert polynomials that 
was developed in [FS, FK1] (see also [FK2, FK3]); this approach is based on an 
exponential solution of the Yang-Baxter equation that is related to the nilCoxeter 
algebra of the symmetric (A n case) or the hyperoctahedral (B n case) group. On 

(n) 

this way, we introduce certain polynomials 23^ which can be viewed as candidates 
for S n -Schubert polynomials. The algebraic-geometry sense of these polynomials is 
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at least obscure. However, they prove to have nice combinatorial properties which 
are similar to those of ordinary Schubert polynomials. In particular: 

(i) they satisfy similar recurrences with divided differences (with a single excep- 
tion that corresponds to the "special" generator sq of the hyperoctahedral group 

with generators so, ... , s n _i); 

(ii) they have direct combinatorial interpretation in terms of "_B n -braids" ; it 
means, in particular, that they have nonnegative coefficients; 

(iii) this interpretation can be restated in terms of reduced decompositions of w 
and "compatible" sequences; 

(iv) the defining expression in the nilCoxeter algebra has n 2 factors; these factors 
are in natural bijection with the entries of a standard reduced decomposition of wq, 
the element of maximal length in W^ n >; 

(v) polynomials 05™ have a stability property, in the sense that the coefficient 
of any monomial in 03^ gets fixed when n is sufficiently large; 

(vi) we introduce symmetric functions H w which have a similar combinatorial 

(n) 

interpretation and can be obtained as a certain limit of *Bw s; these functions 
generalize Schur P-functions. 

This paper is organized as follows. Section 2 contains a straightforward adap- 
tation for the B n case of the main "geometric" construction used in [FK1]; the 
role of the Yang-Baxter equation (YBE) is briefly explained. (At this point, an ac- 
quaintance with our "^"-paper [FK1] would be very helpful.) In Section 3, some 
exponential solutions of S n -YBE are listed (we refer to [FK3] for details). Section 
4 introduces 5 n -symmetric functions which one can associate with any such solu- 
tion. Generalized Schubert expressions for the B n case are suggested in Section 5. 
For the nilCoxeter algebra solution, these expressions give rise to polynomials 
which are studied in Section 6. Section 7 contains the full list of 93^ 's for n < 3. 
Section 8 is devoted to £? n -analogues of stable Schubert polynomials. 

Acknowledgements. The authours are grateful to Richard Stanley for helpful 
comments. 

2. Generalized configurations and the Yang-Baxter equation 

The notion of generalized configuration was introduced in [FK1]. It is a configura- 
tion of contiguous lines which cross a given vertical strip from left to right; each line 
is subdivided into "segments" ; each segment has an associated variable. A config- 
uration is assumed to be generic in the following sense: (i) no three lines intersect 
at the same point; (ii) no two lines intersect at an endpoint of any segment; (ii) no 
two intersection points lie on the same vertical line. 

In the B n case, this notion assumes an additional flavour. Namely, configurations 
are contained in a semi-strip bounded from below by a bottom mirror (cf. [Ch]). 
The lines of a configuration are allowed to touch the bottom; whenever it happens, 
the associated variable changes its sign. An example of a S n -configuration is given 
on Figure 1. 

We are particularly interested in intersection points and points of reflection. Each 
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this point (the point itself contributes 1). For example, the intersection points on 
Figure 1 have level numbers (from left to right) 1, 1, 2, and 1. The level number of 
a point of reflection is by definition. 

Let C be a configuration of the described type. Let us order its intersection and 
reflection points altogether from left to right; then write down their level numbers. 
The resulting sequence of integers a(C) = a\a 2 ... is called a word associated with 
C. In our running example, a(C) = 101201. Now it is time to bring the variables 
into the picture. Assume that A is an associative algebra and {hi(x) : i = 0, 1, . . . } 
is a family of elements of A which depend on a formal variable x (we always assume 
that the main field contains all participating formal variables) . Then the associated 
expression for a configuration C is 

$(C) = h ai ( Zl ) h a2 {z 2 ) ... 

where, as before, aia 2 ... is an associated word and is one of the following: if 
ai = 0, then is the variable related to the corresponding point of reflection (to 
the left of it); if > 0, then z, = Xi — yi where Xi and yi are the variables for 
the segments intersecting at the corresponding points, Xi being associated with the 
segment which is above to the left of this point. In the example of Figure 1, 

$(C) = $(C;x,y,z,u,v) = hx(y - u)h Q {y)h 1 {v + y)h 2 (x + y)h (v)h 1 (x + v) . 

Informally, the variables associated with the segments are their "slopes" ; an argu- 
ment of each factor in $(C) is the corresponding "angle of intersection". 

The Yang-Baxter equations (see, e.g., [Ch] and references therein) are certain 
conditions on /j,j(x)'s which allow to transform configurations without changing 
their associated expression. In the case under consideration (i.e., in the B n case), 
the YBE are 

(2.1) hi{x)hj(y) = hjiy^iix) if \i - j\ > 2 ; 

(2.2) hi(x)h i+1 (x + y)hi(y) = h i+1 (y)hi(x + y)h i+1 (x) if i > 1 ; 

(2.3) hi{x - y)h (x)h 1 (x + y)h (y) = h (y)h 1 (x + y)h (x)h 1 (x - y) . 

Each of these equations has its pictorial interpretation; see Figure 2. 
Following [FK1], we introduce the additional condition 

(2.4) hi{x)hi{y) = hi{x + y) , ^(0) = 1 if z > 1 

(cf. Figure 3). Conditions (2.1)-(2.4) have various nice implications. To derive 
them, one can use "geometric" techniques instead of cumbersome algebraic com- 
putations. Informally, algebraic identities can be proved by moving lines according 

±„ j-l 1„„ „c T?;„ „ o o 
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3. Examples of solutions 

There is a natural way to construct solutions of the equations (2.1)-(2.4). Assume 
A is a local algebra with generators uq, u±, U2, ... which means that 

(3.1) UiUj = UjUi if \i — j\ > 2 . 
Define hi(x) by 

(3.2) hi(x) = exp(xui) ; 

then (2.1) and (2.4) are automatically guaranteed, and we only need the Yang- 
Baxter equations (2.2)-(2.3) to be satisfied. Rewrite (2.2)-(2.3) as 

(3 3) e xui e (x+y)u i + 1 e yui _ e yu i + 1 e (x+y)ui e xu i+1 

and 

(3.4) { e xuo e xu le xuo^ e yuo e yu le yu } = Q _ 

These equations were studied in [FK3] where the following solutions were suggested. 

3.1 Example. The nilCoxeter algebra of the hyperoctahedral group. This is the 
algebra defined by 

UiUj = UjUi , \z-j\ > 2; 
^ = 0; 

UiU i+1 Ui = u i+1 UiU i+1 , % > 1 ; 

UqUiUqUi = UiUqUiUq ■ 

3.2 Example. Universal enveloping algebra of U + (so(2n + 1)). This algebra can 
be defined as the local algebra with generators u±, U2, ... subject to Serre relations 

[ui, [ui,Ui±i]] = ,i > 1 ; 
[u , [uq,^]] = ; 
[u u [ui,u ]]] = . 

Example 3.1 will be the main one in this paper. Note that the nilCoxeter algebra 
can be alternatively defined as the algebra spanned by the elements of a Coxeter 
group, with the multiplication rule 



w ■ v = 



usual product wv if l(w) + l(v) = l(wv) 
, otherwise 



where l(w) is the length of w (the minimal number of generators whose product is 
...\ 
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4. Symmetric expressions 

By analogy with [FK1], we will show now that the basic relations (2.1)-(2.4) (or 
(3.1)-(3.4)) imply that certain configurations produce symmetric expressions in cor- 
responding variables. In what follows we assume that (2.1)-(2.4) are satisfied. 

4.1 Theorem. For the configuration C of Figure 4, 

$(C;x,y) = &(C;y,x) . 

In other words, $(C) is symmetric in x and y. 

This statement has the following obvious reformulation. 

4.2 Proposition. Let 

(4.1) B(x) = h n -i(x) ■ ■ ■ hi(x)h (x)h 1 (x) ■ ■ ■ h n -i(x) . 

Then B(x) and B(y) commute. 

Special case: n = 2. Then B{x) = h 1 (x)h (x)hi(x). Now use (2.3) and (2.4) 
to show that 

B{x)B{y) = h 1 (x)h (x)h 1 (x)h 1 (y)h (y)h 1 (y) 

= h 1 (y)h 1 (x - y)h (x)h 1 (x + y)h (y)h 1 (y) 
= h 1 (y)h (y)h 1 (x + y)h (x)h 1 (x - y)h 1 (y) 
= h 1 (y)h (y)h 1 (y)h 1 (x)h (x)h 1 (x) 
= B{y)B{x) . 

The same proof can be performed in the language of configurations — see Figure 

5. Moreover, the geometric proof has the advantage of being easily adjustable for 
the general case of an arbitrary n. 

Proof of Theorem 4.1 (and Proposition 4.2). Same transformations as in 
Figure 5, with additional horizontal lines added near the bottom. □ 
Proposition 4.2 implies that the expression 

(4.2) H^(x 1 ,x 2 ,...) = B(x 1 )B(x 2 )-- 

is symmetric in Xj's. Therefore one can construct examples of symmetric functions 
by taking any solution of (2.1)-(2.4), then any representation of the corresponding 
algebra A, and applying the operator representing H^ n \xi, x 2 , . . . ) to any vector. 

Surprisingly enough, the expression H^ n \x\, x 2 , ■ ■ ■) can be alternatively defined 
by a quite different configuration. 

4.3 Theorem. Let C be the configuration defined by Figure 6. Then 

<$>(C;x ll ...,x n ) = H M (x u ...,x n ) . 

t» e o — t?;„ t i — i 
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Remark. If the number of generators is m > n, then, to get H^ m >(xi, . . . , x n ), one 
only needs to add m — n horizontal lines near the bottom mirror to the configuration 
of Figure 6. 

Theorem 4.3 allows us to relate the B n - and ^-constructions to each other. 
Note that the configuration of Figure 6 coincides with one of [FK1, Figure 14] up 
to renumbering the variables x\ in the opposite order (this is not essential since 
the expression is symmetric in Xj's), setting yi = —Xi, and dropping (sorry) the 
bottom mirror. Since Figure 14 of [FK1] defines the ordinary (i.e, S n -) double 
stable Schubert expression G(xi, . . . , x n ] y±, . . . , y n ), the above observation has the 
following precise formulation. 

4.4 Theorem. Let {hi(x) : i = 1, . . . , n — 1} be any solution of (2.1), (2.2), 
and (2.4); in other words, let {hi(x)} be an exponential solution of the A n -\-YBE. 
Define ho(x) = 1. Then (2.3) obviously holds and so is well-defined. Moreover, 
in this case 

^ (x \ , . . . , x n ) G(x\ , . . . , X ni X\ , . . . , Xn) 

where G(. . .) is the double stable Schubert expression (see [FK1]). 

In the special case of the nilCoxeter solution of Example 3.1 we obtain B n - 
analogues of Stanley's symmetric functions [S] (stable Schubert polynomials) . These| 
functions are studied in Section 8. 

5. Schubert expressions 

Define the _B n -analogue of the generalized Schubert expression by 

(5.1) <B (n) (xi, . . . , X n ) = (xi, . . . , X n )e(- Xl , -X n -l) 

where &(x\, . . . , x n -i) is the A n _i-Schubert expression as defined in [FS, FK1]. In 
other words, 

(5.2) <B^0n, . . . ,x n ) = B( Xl ) ■ ■ ■ B(x n )A 1 (-x 1 ) ■ ■ ■ A n -i(-x n -i) 
where 

(5.3) Ai(x) = hn-ihn-2 ■ ■ ■ hi(x) 

(recall that B(x) is defined by (4.1)). 
The formula (5.2) can be simplified. 

5.1 Theorem. ^ n \xi, . . . ,x n ) is equal to the expression defined by Figure 8. In 

other words, 

(5.4) 

n — 1 / n—i—1 \ 

<8 (n) (xi,...,x n ) = 6(x„,x„_i,...,x 2 ) n ( h (x n _i) ] J hj(x n -i-j + x n -i)) 
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where, as before, &(x n ,x n -i, . . . , X2) = Ai(x n ) ■ ■ ■ A n _i( X 2) and in the products 
Y[ - • • the factors are multiplied left-to-right, according to the increase of i and j, 
respectively. 

Note that the total number of factors . . ) in (5.4) is n 2 , the length of the 
longest element w of the hyperoctahedral group W^ n > with n generators. Moreover, 
it can be immediately seen from Figure 8 that these factors are in a natural order- 
respecting bijection with the entries of the standard reduced decomposition of wq: 

n — 1, n — 2, 2, 1, n — 1, n — 2, 2, 1, , n — 1, n — 2, 2, 1. 

5.2 Examples. 

n=l <BW( Xl ) = h (x 1 ) 

n = 2 <B (2) (xi, x 2 ) = h 1 (x2)h (x 2 )h 1 (x 1 + x 2 )/io(^i) 
n = 3 %$( 3 \xi, x 2 , x 3 ) 

= h 2 (x 3 )h 1 (x 3 )h 2 (x 2 )h (x 3 )h 1 (x 2 + x 3 )h 2 ( Xl + x 3 ) h (x 2 ) h 1 (x 1 + x 2 )h (. 

Proof of Theorem 5.1. Let $6 and be the expressions defined by configu- 
rations of Figures 6 and 8, respectively. Then 

$ 6 = $ 8 i n _i(x n _i) • • • A 2 {x 2 )A 1 {x 1 ) 

where 

Ai(x) = hi(x)h i+1 (x) ■ ■ ■ h n -x{x) . 
Since (2.4) implies that (A^x)) -1 = Ai(—x), it follows from Theorem 4.3 that 

$ 8 = §6Ai(-x 1 )A 2 (-x 2 ) ■ ■ ■ A n _i(-X n _i) 

= i7 (n) (xi, . . . ,X n )&(-Xi, -X n -l) 

= »< n >(z 1 ,...,x n ) . □ 
6. B n -analogues of Schubert polynomials 

In the rest of this paper we study the main example of solution of (2.1)-(2.4), 
namely, the one related to the nilCoxeter algebra of the hyperoctahedral group 
(Example 3.1). In this example, hi(x) = 1 + xui where ui is the i'th generator. As 
in the case of the symmetric group (cf. [KF1]), we define the S n -analogues of the 
Schubert polynomials by expanding corresponding expression in the basis of group 
elements: 

(6.1) QS^On, ...,*„)= J2 %tt ) (xu---,xn)w; 

wew (n ) 

similarly, the corresponding symmetric functions (stable Schubert polynomials) are 
defined by 



(6.2) 



H^( Xl ,...,x k )= H£\x 1 ,...,x k )w 
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These definitions can be straightforwardly restated in terms of reduced decom- 
positions and "compatible sequences" (cf. [BJS, FS]). Use (4.1)-(4.2) to rewrite 

(6.2) as 

(6.3) H^(x u ...,x k )= E 2^x bl x b2 ---x bl 

ai...a(efl(w) l<6i<---<b(<fc 

ai<a i+1 >a i+ 2 => bi<b i+ 2 

where R(w) is the set of reduced decompositions of w and 

7 (a,b) = #{6 i }-#{i : a, = 0} 

(here denotes the number of different entries in the sequence b±, ... ,6;). 

Correspondingly, (5.1) can be presented as 

(6.4) < 3^(x 1 , ...,x n )= Hi n \ Xl ,..., x n )e v (-x u -* n _i) 

uv=w 
l(u)+l(v)=l(w) 

»eA„_i 

where, as before, & v is the ordinary Schubert polynomial for A n _i = S n . It is 
also possible to entirely rewrite in terms of reduced decompositions and compatible 
sequences the definition of Theorem 5.1. We avoid doing that since the resulting 
formulas are rather messy; we also think that the following "geometric" approach 
(cf. [FK1, Sec. 6]) is more natural. 

Both and have a direct combinatorial interpretation in terms of "re- 
solved configurations"; this interpretation can actually be applied to any polyno- 
mials which come from any configuration C. Take all the intersection points of C 
and "resolve" each of them either as x or as x . Then take all points of reflection 
and resolve each of them either as ^ or as V ; the latter corresponds to changing 
a "sign" , or a "spin" , of corresponding string. If a configuration has iV intersection 
and reflection points altogether, then there are 2 N ways of producing such a reso- 
lution. Each of the 2^ resolved configurations is a "signed braid" which naturally 
gives an element w of the hyperoctahedral group. Reading the x- and V-points 
from left to right produces a decomposition of w into a product of generators. Let 
C w , for a given u>, denote the set of resolved configurations which give w and for 
which this decomposition is reduced. Then the polynomials <& w associated with C 
(that is, 

$(C;x 1 ,x 2 , ■ ■ •) = ^2® w (x 1 ,x 2 , . . •) w ) 

w 

can be expressed as 

(6.5) <m*i,*2,...)= E((n^-^))(n^)) 

where the first product is taken over all intersections in c and the second one - 
over all "change-sign" (i.e., V-) points. 

This interpretation enables us to prove some stability properties of %$w and 

zr(n) 
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6.1 Theorem. Let and W^ m \ n < m, be the hyperoctahedral groups with 

generators s±, ... ,s n and s\, ... ,s m , respectively. Then, for any w G C 

(6.6) »&° (x 1 ,...,x n ) = «B^ m) (xi,...,x n ,0,..., 0) 



and 

(6.7) HW(x 1 ,...,x k ) = HW(x lt ...,x k ) . 

In other words, 's are stable (so we may drop superscripts) and 03™ 's 

are not - - but their coefficients are. Indeed, (6.6) means that the coefficient of 
any monomial in 93™ (xi, . . . , x n ) stabilizes as n — > oo. This allows to introduce 
well-defined formal power series 

(6.8) ?8 tt ,(xi J X2,...)= lim 5S^)(xi,...,a; n ) 

n^oo 

which could be viewed as a stable £? n -analogue of ordinary Schubert polynomials. 

Proof of Theorem 6.1 It suffices to prove the case m = n + 1; the general 
statement follows by induction. Let C^ n+1 ^ be the defining configuration for 
or H( n+1 \ Since w G and the words in d n+1 ' ) are reduced, none of them may 

contain the last generator s n . In other words, all the intersections at the uppermost 
level of £( n+1 ) have to be resolved as x. Then the upper string of the resulting braid 
does not contribute anything (cf. (6.5)); thus we get the same result as if we started 
with the configuration obtained from C( n+1 ) by taking out its upper boundary. In 
the case of H^ n+1 \ this modified configuration is exactly the one which produces 
H^ n \ and (6.7) follows. In the case of the modified configuration, with 

x n = 0, is given on Figure 9 where it is transformed to the one of 03 ^ by taking 
out the segment that meets other ones at zero "angles" only. □ 

Divided differences. Recall that the divided difference operator di is defined by 

(6.9) difixu ...) = • / ' r '---- 1 fi ; 

X% Xi-\-\ 

note that the denominator might well be a^+i — xi that would suit us better in 
what follows; however, we keep the standard notation. 

6.2 Theorem. For any w G andi>l, 



(6.10) 



_ di &n) = { ^ if KwSi) = 1(W) + 1 

otherwise 



(Unfortunately, (6.10) is false for i = 0.) 

Proof. As before, let u\ be generators of the nilCoxeter algebra. Then the 
theorem is equivalent to the identity 



in) 
w 
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which, in turn, can be rewritten as 

-dM n) = & n) Ui ; 

the latter follows from (5.1) and the identity di& — &Ui (see [FS, Lemma 3.5]) which 
is just another way of stating the divided-differences recurrence for the ordinary 
Schubert polynomials. □ 

Note that a result analogous to Theorem 6.2 holds for the power series *B W 
defined in (6.8). 

7. Computations 

Polynomials 58^ can be computed by expanding (5.4) in the basis of permutations. 
One can also use (6.3)- (6.4) and/or the recurrences (6.10). We give below the results 
of our computations for n = 1, 2, and 3; we used the formulas of Example 5.2 and 
then double-checked the results with (6.10). 



n = 1 



<B(i)( Xl ) = hofa) = 1 + x 1 u 



w 



<B (1) 



1 

Uo 



1 
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n = 2 

& 2 Xx u x 2 ) 

= h 1 (x2)h (x 2 )hi(xi + x 2 )h (x 1 ) = (1 + x 2 ui)(l + x 2 u )(l + (x± + x 2 )ui)(l + £i«o)| 
w ®l 2) 



1 1 

Uq Xi + X 2 

Ui x\ + 2x 2 

U Ui x 2 (xi+x 2 ) 

UiU (xi+x 2 ) 2 

UqUiUq XiX 2 (xi+X 2 ) 

uiUqUi x\{x\ + x 2 ) 

wq xixl(xi + x 2 ) 



n = 3 

^ 3 \xi,x 2 ) = h 2 (x 3 )h 1 (x 3 )h 2 (x 2 )h (x 3 )h 1 (x 2 + x 3 )h 2 (xi + x 3 )h (x 2 )hi(xi + x 2 )h (xi) 
= (1 + x 3 u 2 )(l + X 3 «i)(l + x 2 u 2 )(l + x 3 u )(l + (x 2 + x 3 )ut) 

x (1 + (x! + x 3 )u 2 )(l + x 2 u )(l + (xi + x 2 )u 1 )(l + XtUo) I 

In the following table, L = xi + x 2 + x 3 , K = (xi + x 2 )(x\ + x 3 )(x 2 + x 3 ). 



123 1 11 

T23 U 1 L 

213 ui 1 x\ + 2x 2 + 2^3 

132 u 2 1 xi + x 2 + 2x 3 

213 uqU\ 1 (x 2 + x 3 )L 

132 uqu 2 2 (xi + x 2 + 2x 3 )L 

213 U!U 1 L 2 

231 uiu 2 1 2x 3 L + x\x 2 

312 u 2 u x 1 L 2 + (x 2 + x 3 ) 2 

213 UqUiUq 1 K 

23T u uiu 2 1 x 3 (xi + x 3 )(x 2 + x 3 ) 

312 u u 2 ui 2 (x 2 + x 3 )(L 2 + x 2 x 3 ) 

123 uiUqUi 1 (x 2 + x 3 )(x 3 L + x\ + xix 2 ) 

231 uiUqu 2 2 x 3 L 2 + K 

321 u\u 2 u\ 2 K + x\x\ + x 3 {x\ + 3x 2 + 2x 3 )L 

312 u 2 uiUq 1 L{x\ + x\ + x\ + xix 2 + x t x 3 + x 2 x 3 ) + xix 2 x 3 

123 uqUiUqUi 2 (x 2 + x 3 )K 
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32T 


U0U1U2U1 


3 


x 3 (x 2 + x 3 )(xi + x s )(xi + 2x 2 + X3) 


312 


uqu 2 uiUq 


2 


KL 


132 


UiU UiU 2 


1 


xl(x 2 + x 3 )(x 1 + x 3 ) 


321 


UIUQU2UI 


2 


(x 2 + x 3 )K + x 2 x 3 (x 2 + x 3 )L 


321 


U1U2U1U0 


3 


x\L + KL + x 3 (xi + X2)(L 2 — xix 2 ) 


132 


U 2 UiUqUi 


1 


(x 2 + x 3 )(x\L + x\L + xxx 2 x 3 ) 


132 


UqUiUqUiU 2 


3 


x\K 


32T 


UqUiUqU 2 Ui 


2 


x 2 x 3 K 


32l 


U0U1U2U1U0 


5 


x 3 KL 


132 


UQU2UIUQU\ 


3 


K(x\ + x 2 x 3 + xf) 


312 


U\UqU\U2Ui 


3 


xl(xi + 2x 2 )(xi + x 3 )(x 2 + x 3 ) 


321 


U1UQU2U1UQ 


2 


K(xix 2 + XiX 3 + x 2 x 3 ) 


231 


UiU 2 U\UqU\ 


3 


x 2 x 3 (x 2 + x 3 )(x 3 L + x\ + xix 2 ) + K(x\ + x 2 x 3 + x\) 


123 


U 2 UiUqUiU 2 


1 


xl(x 2 +x 3 )(x 1 + x 3 ) 


312 


UqU\UqUiU 2 U\ 


5 


X2X\K 


321 


U UiU U2UiU 


2 


xix 2 x 3 K 


231 


U0U1U2U1U0U1 


5 


x 2 x 3 (x 2 + x 3 )K 


T23 


UqU 2 UiUqUiU2 


4 


xlK 


312 


UiUqU\U 2 U\Uq 


5 


xl(xi + x 2 )K 


231 


U\UqU2UiUqU\ 


5 


(x2X 3 L + x\x\ + x\x\)K 


213 


U1U2U1UQU1U2 


4 


x\{x\ + 2x 2 )(xi + x 3 )(x 2 + x 3 ) 


312 


UqU\UqUiU2UiUq 


7 


X\X2X\K 


231 


UqU\UqU 2 U\UqU\ 


7 


xix 2 x 3 (x 2 + x 3 )K 


213 


U0U1U2U1U0II1U2 


9 


x 2 x 3 K 


132 


U1U0U1U2U1U0U1 


5 




213 


U1UQU2U1UQU1U2 


9 


(xi + x 2 )xlK 


132 


UqU\UqUiU2UiUqUi 


12 


x\x\x\K 


213 


UqU\UqU 2 U\UqU\U2 


16 


X\X2X\K 


123 


U\UqUiU2UiUqU\U2 


14 


x\x\K 


123 


UqU\UqUiU2UiUqUiU2 


42 


x\x\x\K 



8. Symmetric functions 

This section is devoted to studying the basic properties of the symmetric functions 
H n defined by (6.2), (4.2), and (4.1). 

In the nilCoxeter case, Theorem 4.4 immediately allows to establish the follow- 
ing connection between H w : s and Stanley's symmetric functions (stable Schubert 
polynomials) G w . 

8.1 Corollary. Let x = (xi, . . . ,x n ). Let w be an element of the symmetric 
subgroup of the hyperoctahedral group W^ n ' that is generated by s\, ... , s n -i- 
Then 

^(x)=C er (x,x) 
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The last formula implies that, for such w, H w is a nonnegative integer linear 
combination of Schur P-functions. Also, the definition of the functions H w implies 
that they are, for any w, some linear combinations of P-functions. Tao Kai Lam 
recently found a proof that, in fact, H w is always a nonnegative integer linear 
combination of P-functions. 

It can be shown that the [skew] P-functions themselves are a special case of H w 's. 
To do that, we generalize, in a more or less straightforward way, the corresponding 
^-statement about 321-avoiding permutations (see [BJS]). 

8.2 Theorem. Let a be a skew shifted shape presented in a standard "English" 
notation (see, e.g., [SS]). Define a "content" of each cell of a to be the difference 
between the number of column and the number of row which this cell is in. (E.g., 
the content of a cell lying on the main diagonal is 0.) Read the contents of the 
cells of a column by column, from top to bottom; this gives a sequence a\, ... , a\. 
Define an element w a of the hyperoctahedral group by w a = s ai • • • s ai (in 
fact, a\, . . . , ai is a reduced decomposition of a). Then H Wa = P a where P a is the 
skew Schur P -function corresponding to the shape a. 

One could also ask: which elements w G can be presented as w a (see 

Theorem 8.2)? The answer can be presented in either of the following equivalent 
ways (both are informal though unambigous): 

(i) those w for which the set of reduced decompositions R(w) contains a single com- 
mutation class (i.e, they can be obtained from each other by switches of commuting 
variables, without any use of Coxeter relations); 

(ii) those w which avoid the following patterns: 

321 321 32T 3 2 1 12 12 

where i denotes the element i of a signed permutation that has changed its sign. 

Technical proofs of the last statement and of Theorem 8.2 will be given elsewhere. 
H w as a limit of 23^. Similarly to the S n case, the symmetric functions Hw can 
be obtained as some kind of limit of 23^ n ' ) as m — > oo. 

8.3 Theorem. Let w be any element of the hyperoctahedral group with n genera- 
tors. Then 

lim^ +N \0 1 _ : ^0 1 x ll ...,x ri ) = H^(x u ...,x n ) . 

N 

Even more: if N > n — 1, then 

^ (0, . . . , 0, xi, . . . , x n ) = ^ (xi , . . . , x n ) . 
Proof. Similar to the proof of Theorem 6.1. □ 
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